DOCUMENT RESUME 



ED 453 438 



CE 081 862 



TITLE 
PUB DATE 
NOTE 



EDRS PRICE 
DESCRIPTORS 



PUB TYPE 



IDENTIFIERS 



Evaluation in HRD. Symposium 29. [AHRD Conference, 2001]. 
2001 - 00-00 

26p.; In; Academy of Human Resource Development (AHRD) 
Conference Proceedings (Tulsa, Oklahoma, February 28-March 
4, 2001) . Volumes 1 and 2; see CE 081 829. 

Collected Works - General (020) -- Speeches/Meeting Papers 

(150) 

MF01/PC02 Plus Postage. 

♦Corporate Education; Education Work Relationship; 
Educational Needs; *Evaluation Methods; *Evaluation 
Utilization; Human Resources; Influences; *Labor Force 
Development; Learning Readiness; Needs Assessment; 
♦Organizational Climate; Organizational Development; 
Postsecondary Education; ♦Program Evaluation; Research 
Methodology; Surveys; Training Objectives 
Impact Studies; Stakeholders 



ABSTRACT 



This document contains three papers on evaluation in human 



resource development (HRD) . "Utilization of Corporate University Training 
Program Evaluation" (Christopher F. Bober) reports on a study that examined 
how training program evaluation results are being utilized in corporate 
universities and determined that the following seven factors affect how four 
corporate universities are utilizing program evaluation data; communication 
quality; timeliness; commitment and/or receptiveness to evaluation; 
evaluation quality; credibility; relevance; and findings. "Organizational 
Readiness for Learning and Evaluation" (Hallie Preskill, Brenda 
Martinez - Papponi , Rosalie T. Torres) discusses the outcomes of implementation 
of a diagnostic instrument designed to assess organizations ' readiness for 
learning and evaluation . The instrument was used with eight U. S . 
organizations and found to be capable of identifying areas of strength on 
which to build evaluation efforts and areas in need of development for 
creating and sustaining learning from evaluation work. "Three Perspectives of 
Training Evaluation Based on Organizational Needs" (Eul-Kyoo Bae, Ronald L. 
Jacobs) argues that HRD professionals must place more emphasis on matching 
the intents of evaluation, the intended users, and aspects of the evaluative 
process and information. Three perspectives of stakeholders on evaluation of 
training were developed as a means for ensuring the match between evaluation 



intents and activities. All three papers include substantial bibliographies. 
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This study aimed to determine how training program evaluation results are being utilized in corporate 
universities. In addition, this study also attempted to identify influential factors that affect evaluation data 
utilization in these entities. The majority of the purposes identified in this study were instrumental in 
nature. In addition, several examples of both conceptual and symbolic use were also cited by participants. 

Seven major factors were found to impact the use of evaluation results in the four corporate universities. 

Keywords: Evaluation Use, Corporate University, Training Program Evaluation 

Throughout the United States, businesses are spending increasing amounts of money each year on the training and 
development of their employees. The Bureau of Labor Statistics estimates that in 1997, businesses spent 58 billion dollars on 
training. Training Magazine’s “Industry Report” states that, in 1997, employees in United States businesses received 751 
million hours of training (Industry Report, 1997). It is clear that organizations are recognizing the importance of training their 
employees in order to help their businesses operate as productively and profitably as possible. 

Many organizations are going one step further in the training and development of their human resources and are 
establishing what is known as a “corporate university.” A corporate university is essentially a separate entity in a company 
that is responsible for the development and implementation of training programs for the members of the organization 
(Meister, 1994). It can be assumed that these organizations that have made an investment to develop a corporate university 
have made training a top priority. 

With the increase of time, money, and other resources that are being invested in training by United States 
companies, the need for comprehensive evaluation techniques that measure the effectiveness of these training efforts is 
becoming more important than ever before. The training and development literature continually stresses the importance of 
evaluating training. However, among the companies that do use some form of evaluation technique, only a small percentage 
actually understand the importance and procedure of not only conducting the evaluation, but then utilizing the evaluation 
results correctly (Robinson & Robinson, 1989; Camevale & Schulz, 1990; Dixon, 1990; Phillips, 1991). 

The literature surrounding training evaluation examines the importance of conducting sound evaluations and 
suggests numerous reasons about why it is important to evaluate training programs. Moreover, within this literature, the issue 
of utilizing the evaluation results is regularly discussed. Evaluation can be used for a number of purposes. However, using 
evaluation for the improvement of training programs (formative evaluation) and for accountability, justification, and program 
continuation purposes (summative evaluation) are the most common reasons referred to throughout the training literature 
(Worthen, Sanders, & Fitzpatrick, 1997). 

For evaluation to actually serve either a formative or summative purpose, the results of the evaluation must be 
utilized. Although there are a number of components involved in the evaluation of training programs, the utilization of 
evaluation results is continually noted throughout the literature as being the most important evaluation component and the 
most often overlooked (Wentling, 1980; Piskurich, 1997; Robinson & Robinson, 1989). 

Training departments most often utilize the results of evaluations to aid in meeting training program, employee, and 
management demands (Camevale & Schulz, 1990). They use evaluation results in a variety of ways to assist in improving 
their trainin^^ nmcrrams. Trainint^ Henartmenus use the results to dernonstrate a oro<yram’s worth or value tn the organization. 
Training managers and employees use evaluation results to help establish whether the employees have learned the necessary 
knowledge, skills, and attitudes to perform their specific jobs. Upper-level management want to know how the training 
department and its programs are affecting the company’s bottom line. Training departments can use the results of evaluations 
to justify training expenditures with regard to return on investment, return on sales, and customer satisfaction (Camevale & 
Schulz, 1990). 

Since the mid 1970’s, the majority of studies that examined the utilization of evaluation results looked primarily at 
the different factors that influenced the use. In addition, many of these studies also focused their attention on defining what 
was meant by “evaluation use.” 

From the research conducted throughout the 1 970’ s, it was determined that evaluation use could be either 
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instrumental, conceptual, or symbolic (persuasive) in nature. Rich (1977) explained instrumental use occurred when 
evaluation results were used in a direct and immediate manner, most often for program improvement. 

Conceptual use occurred when evaluation findings were utilized overtime to influence thinking and future decision- 
making (as opposed to being used for an immediate and observable action). Symbolic or persuasive use referred to the 
utilization of evaluation results to help justify a program and serve as an accountability mechanism or to give individuals a 
political advantage against opponents (Leviton & Hughes, 1981). 

In 1977, Patton, Grimes, Guthrie, Brennan, French, and Blyth conducted a study where they interviewed decision- 
makers and evaluators from twenty federal health projects. They focused their research on identifying factors that affected the 
use of evaluations in these projects and the extent to which the evaluation results were used. 

In attempting to identify the key factors that influence evaluation results utilization, the researchers used a list of 
eleven common factors that had been determined by previous studies. They asked the participants to comment on the overall 
importance of each of the eleven factors. From the eleven factors, a total of two major factors emerged as being consistently 
relevant in helping to explain the utilization (or non-utilization) of evaluation results. These factors were identified as political 
factors and personal factors (Patton, et al., 1977). 

From the twenty cases that were studied, the researchers found that in fifteen of them, the participants believed that 
politics played a major role in the utilization process. These political factors included: intra- and inter-agency rivalries 
(budgetary disputes with the Office of Management and Budget, the administration, and Congress); power struggles between 
Washington administrators and local program personnel; and internal debates about the purpose and/or accomplishments of 
programs (Patton, et al., 1977). 

The personal factor(s) that emerged as being of significance in the utilization process were comprised of seven parts 
which included: leadership, interest, enthusiasm, determination, commitment, aggressiveness, and caring (Patton, et al., 
1977). The researchers indicated that “evaluations have an impact when personal factors are present, and an absence of 
impact when not presenf ’ (Patton, et al., 1977). 

Patton, et al. (1977) stressed throughout their study that the evaluator's dedication to the evaluation project had great 
influence over whether the results would be utilized. In addition, Patton also found that the more the evaluative information 
was directed at advocates of the program being evaluated, the greater the likelihood that they would use the results (Leviton & 
Hughes, 1981; Patton, etal., 1977). 

In 1997, Preskill and Caracelli (1997) conducted survey research focused on evaluation use that analyzed 
perceptions and experiences about use. They used an informed sample population that consisted of members of the 
Evaluation Use Topical Interest Group. The researchers found that the strategies identified most by the participants as helping 
to promote use included working collaboratively with clients and stakeholders throughout the evaluation, effectively 
communicating the results of an evaluation to the proper audience(s), and establishing an effective method for drawing 
conclusions based on the results which would ultimately lead to making decisions for action based on this communicated 
information (Preskill & Caracelli, 1 997). 

Problem Statement 

This study examined the utilization of training program evaluation results in corporate universities. Specifically, this study 
attempted to determine the factors that are related to the use of evaluation results in corporate universities. 

Research Questions 

This study was guided by the following four research questions: 

1 . What individual(s) within the corporate university and/or organization use the results of the evaluations? 

2. What are the main purposes for which the results of corporate university training program evaluations are used? 

3. What components of the evaluation report are most utilized? 

4. Based on Cousins and Leithwood’s 1986 framework, what are the factors that influence the use of evaluation results in 
corporate university training programs? 

A. What is the applicability of the Cousins and Leithwood 1986 meta-analytic conceptual framework to corporate 
university training programs? 

B. Do the factors identified in this framework as influencing evaluation utilization in a number of contexts (i.e., 
education, mental health, social services) have any relationship with the factors that are found to influence use in a corporate 
university training setting? 

C. Which of the higher order categories (and subsequent factors) identified in Cousins and Leithwood’s 1986 
framework have the most impact on decision-makers and the use of evaluation results? 
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Theoretical Framework 



Throughout the evaluation utilization literature, it is evident that the majority of historical research has used frameworks such 
as the one developed by Alkin, et al (1983) to help in identi lying factors that influence the use of evaluation results. Most of 
these early frameworks simply described the factors that influenced the use of results. Until 1986, no comprehensive study 
succeeded in going beyond these existing frameworks to try and create, through a meta-analysis of frameworks, a model that 
built on (and synthesized) the most important components of each of these earlier studies. 

In 1986, Cousins and Leithwood conducted an in-depth investigation into all of the empirical research on the 
utilization of evaluation results (65 total studies) that had taken place since 1971. Cousins and Leithwood (1986) were the 
first researchers to actually construct a meta-analytic model, or conceptual framework, that looked beyond just describing 
influencing factors. Their framework included the most important/significant factors from previous models and could be used 
as described by Shulha and Cousins in 1997, “to determine the weight of the factors in their ability to predict use” (Shulha & 
Cousins, 1997). 

Cousins and Leithwood ’s investigation examined 65 studies that were focused on the use of evaluation results. 
From these studies, the researchers were able to construct a conceptual framework that is made up of 1 2 factors that influence 
use. They grouped six factors under a “higher order category” called evaluation implementation (relevance, timeliness), and 
the remaining six factors under a second “higher order category” called decision/policy setting (political climate, competing 
information) (Cousins & Leithwood, 1986). The dependent variable in their study, and framework, was evaluation 
utilization, while the independent variables were the factors that influenced the ways the results were used by key decision- 
makers (Cousins & Leithwood, 1 986). 

After reviewing the 65 empirical studies. Cousins and Leithwood determined that 12 major factors were common 
throughout the studies and could be grouped under the higher order categories listed above. The factors associated with the 
category of evaluation implementation included: evaluation quality, credibility, relevance, communication quality, findings, 
and timeliness. The remaining factors grouped under the category of decision/policy setting were: information needs, 
decision characteristics, political climate, competing information, personal characteristics, and commitment and/or 
receptiveness to evaluation (Cousins & Leithwood, 1986). 

The framework for conceptualizing evaluation utilization that was designed by Cousins and Leithwood in 1986 
served as the basis for this study. This framework was chosen for this study because of its overall comprehensiveness and 
applicability for examining factors and other aspects of evaluation utilization. This is due to Cousins and Leithwood’s 
detailed, in-depth study of previous models to create, through a meta-analysis of frameworks, a model that built on (and 
synthesized) the most important components identified in earlier studies. 

This investigation tested the Cousins and Leithwood (1986) framework in a corporate training setting. This 
framework was based on a meta-analysis of frameworks that were developed in a number of different settings, and not just a 
corporate training setting. By applying this model to the corporate setting, a comprehensive look into the use of evaluation 
results within corporate university training programs and departments resulted. 

Research Design 

This study was conducted primarily through the use of qualitative multi-site field research. Specifically, this research utilized 
a case study approach, consisting of on-site interviews with key personnel and in-depth document reviews of all applicable 
training program evaluation information at four major U.S. corporate universities. This method was chosen because it 
enables the investigator to examine the general context in which evaluation utilization occurs and provides the best access to 
corporate universities to study the factors that are affecting this use. The combination of on-site, semi-structured interviews 
with individuals at each location and document review of information relevant to each corporate university’s training program 
evaluation, enabled the investigator to utilize iiiultiple data collection methods and data sources which added to the overall 
reliability and validity of the research. In addition, by conducting case study research, the investigator was able to serve as the 
common “interpreter” of all participant responses and was able to assist each participant with any clarification he/she required 
in regard to the interview questions. The investigator had the opportunity to discuss, in detail, all responses with each 
participant, which resulted in a collection of the most in-depth and complete data about how evaluation results are used in 
corporate universities and the factors that affect this use. Qualitative field research enabled the investigator to gain a broad, 
complete, and comprehensive understanding of the different facets that affect the use of training program evaluations. 

Limitations 

The corporate universities used as the population for this study are not representative of all training departments in all 
companies. While the companies who have established corporate universities most likely view the training and development 
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of their employees as being highly important, the data collected from them may not be representative of other companies who 
have not put similar resources into developing a corporate university. Specifically, this study focused on four corporate 
universities identified with the assistance of a jury of experts on corporate universities. 

Results and Findings 

Cross-site Analysis. The cross-site analysis examines the parallels and dissimilarities in the data collected 
from the four corporate universities in this study: Tennessee Valley Authority University (TVAU), Illinova 
University (lU), Bell Atlantic Training, Education, and Development (BA), and Patent and Trademark Office 
University (PTOU). Specifically, this section discusses the relationships evident in the findings from each site with 
regard to the following: users of evaluation information; components of evaluation reports used; uses of evaluation 
data (purposes); and influential factors associated with evaluation utilization. The researcher conducted a total of 
twenty-nine interviews in this study with individuals who serve in a variety of capacities including evaluators, 
instructional designers, deans or directors of corporate universities, instructors/trainers, advisors, and managers from 
throughout the organizations. In addition to these in-depth interviews, the researcher spoke with a number of other 
corporate university staff members (ranging from administrative staff to statisticians to marketing personnel) from 
each site who were able to contribute information used to achieve this study’s objectives. 

Although each of the four sites represent different industries and therefore vary slightly in their training programs’ 
content, their overall curriculums were similar. Each of the corporate universities’ training curriculums were developed 
around the following four major areas: New Employee Orientation/Organization Orientation; New Employee Technical 
Skills Training/Job Skills Training; Incumbent Soft Skills Training (i.e., customer service, communication skills, problem 
solving, planning, project management) and. Incumbent Technical Skills Training/Job Skills Training. Every site had the 
overall goal to train new employees; and, to keep incumbent workers updated and current in the knowledge, skills, and 
various competencies needed to do their jobs. When asked about different methods of evaluation and types of utilization for 
specific training courses within each of their curriculums, all sites indicated that they do not differentiate between types of 
training in terms of certain courses being evaluated and others that are not. Moreover, every site explained that evaluation is 
undertaken and results used for the same purposes for all of their curriculums’ training programs regardless of the type or 
category of training. 

Users of Evaluation Results. Throughout each of the four corporate universities, the major users of training 
program evaluation results tended to be similar and fall into the following six categories: evaluation staff; 
instructional design and development staff; instructors/trainers; the dean or director of the corporate university; 
upper-level management, typically from the corporate universities’ governing boards; and, student 
advisors/counselors. The purposes in which data were used by each of these groups was similar among the four 
sites; however, it was determined that the ways evaluation results were used differed by position of user. For 
example, instructional design staff tend to use evaluation information for internal training activity course and 
instructor modifications and rarely utilized the data for justification purposes as deans tended to do in presentations 
to boards of directors. Although data were used by different individuals for various purposes, a strong sense of 
camaraderie among the users within the corporate universities existed at all of the sites. 

The one difference that existed among the sites with respect to the users of data, had to do with managers from 
departments throughout the organization receiving and utilizing evaluation information. Tennessee Valley Authority 
University (TVAU), Bell Atlantic Training, Education, and Development, and Illinova University were the only sites that 
distributed data to outside department managers. While TVAU and Bell Atlantic Training, Education, and Development 
indicated a considerable amount of use by external department managers, Illinova University noted that although all managers 
receive data about their employees’ progress in training, only a small percentage actually use the information for specific 
purposes. Due to union pressure, the Patent and Trademark Office University has implemented policy that does not allow for 
trainee evaluation results to be distributed to company department managers. 

None of the sites indicated trainees as major users of evaluation data. In fact, with the exception of TVAU, 
employees do not directly receive evaluation results Irom their training programs. PTOU relies on student advisors while the 
remaining three sites utilize department managers to convey evaluation information to trainees and to implement solutions to 
problems identified in the data. Each site felt that its method was a reliable way to communicate data to students for self- 
improvement and other related purposes. 

Components of Evaluation Reports Utilized. All of the sites in this study consistently used similar reporting 
techniques to communicate evaluation data to users throughout their organizations. The components of the 
evaluation reports were dependent, however, on the type of information that was collected as well as the users’ 



29-1 



6 



learning or processing styles. For example, student reaction and test scores tended to be reported using descriptive 
statistics, charts, and graphs, while return on investment information was typically conveyed through qualitative, 
written formats. Furthermore, the evaluation staff at each site indicated that the reporting method(s) used was highly 
dependent on the end-user and his/her format preference for receiving data as indicated from past experience. When 
the evaluation teams were unaware of the users’ preferred format, they incorporated multiple types of 
communication techniques in the evaluation reports to help increase understanding and utilization of the data. 

Across all of the sites, a common trend was apparent in regard to certain groups utilizing specific parts of the 
evaluation reports. The evaluation staff, instmctional designers, trainers, and advisors tended to use the statistical portions of 
reports that provided a detailed breakdown of each item on the assessments and used means, ranges, standard deviations, and 
other descriptive statistics to illustrate the findings from the evaluations. Because these groups of users are the “core” staff 
directly responsible for all aspects of the training programs including the implementation of appropriate solutions to remedy 
problems that occur, more in-depth data are needed to identify specific problems that exist within courses and among the 
student population. 

The deans or directors of the corporate universities, governing board members, upper-level management, arid 
department managers primarily used components of quarterly and/or semester evaluation reports that contained a wider range 
of reporting techniques (i.e., graphs, charts, executive summaries, etc.). The evaluation teams used multiple reporting 
methods because these reports were being distributed to a larger user group with varying types of learning and processing 
styles. 



Uses of Evaluation Data. This study revealed a number of different ways training program evaluation 
results are utilized by corporate universities. Each of the four sites provided an average of ten examples regarding 
how data are being used at their organizations. The majority of the purposes of use identified in this study were 
instrumental in nature, however, both conceptual as well as symbolic types of utilization were present within each of 
the corporate universities. Table 1 contains a comprehensive summarization of the different ways training program 
evaluation data are being put to use at the four corporate universities, the nature of use (i.e., instrumental, 
conceptual, and/or symbolic), and the number and name of sites that indicated each purpose as being prevalent 
within their organization. The list is comprised of a total of nine different purposes of use that were indicated as 
occurring by at least two of the corporate universities. 



Table 1. How Evaluation Results are Utilized Among the Four Sites 



Purpose of use-results used to: 


Nature of use 


Occurrence/ 
Total sites* 


Modify aspects of course/curriculum 


Instrumental 


4/TVAU,IU,BA,PTOU 


Instructor development/replacement 


Instrumental 


4/TVAU,IU,BA,PTOU 


Justification of training programs 


Symbolic 


4/TVAU,IU,BA,PTOU 


Influence course continuation decisions 


InstmL/Conceptual 


4/TVAU,IU,BA,PTOU 


Identify when advanced course is needed 


Instml ./Conceptual 


3ATVAU,IU,BA 


Determine employee Job placement 


Instrumental 


2/PTOU,BA 


Market programs 


Instrumental 


2/IU,BA 


Identify barriers w/in the org. that impact 
the transfer of training 


Instrumental 


2/IU,BA 


Continue/discontinue contracts with 
external v_endors 


Instrumental 


2/BA.PTQU 



*TVAU - Tennessee Valley Authority University 
lU - Illinova University 

BA - Bell Atlantic Training, Education, & Development 
PTOU - Patent & Trademark Office University 

Influential Factors Associated with Evaluation Utilization. The meta-analytic conceptual framework 
developed by Cousins and Leithwood in 1986 was used to determine influential factors that affect the utilization of 
training program evaluation data at corporate universities. Through the meta- analysis, Cousins and Leithwood 
determined the importance (or influence) of each factor and ranked them accordingly one through twelve. 

A comprehensive list of factors identified at all of the sites is summarized according to rank of importance in Table 
2. Seven of the factors from the Cousins and Leithwood framework were noted as influencing evaluation utilization 
throughout all four of the corporate universities. Their overall rank of importance was determined by taking each factor’s 
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rank from all of the sites and computing an overall average for that factor. For example, communication quality ranked as the 
most important factor at three sites and as second most important at the fourth corporate university. Based on these rankings, 
this factor’s overall average was 1 .25, which categorized it across the four sites as being the number one factor affecting use. 



Table 2. Factors Identified as Being Influential to Evaluation Data Utilization 



Factor 


Overall rank of importance 


Communication Quality 


1 


Timeliness 


2 


Commitment and/or Receptiveness to Evaluation 


3 


Evaluation Quality 


4 


Credibility 


5 


Relevance 


6 


Findings 


7 



Through the analysis of each site’s findings in terms of the following: users of evaluation information; components 
of evaluation reports used; uses of evaluation data; and influential factors associated with evaluation utilization, it has been 
determined that many similarities and common trends exist. The main users of evaluation data at each of the sites typically 
consisted of evaluation staff, instructional design and development staff, instructors/trainers, the dean or director of the 
corporate university, individuals from upper-level management, and, student advisors/counselors. 

The components or parts of the evaluation reports that were utilized was dependent upon the user’s personal 
learning style and format preference. Evaluation results were used across all four sites in this study, for nine major purposes 
summarized in Table 1. The majority of use occurred in an instrumental sens^ however, some cases of conceptual and 
symbolic utilization were also noted by study participants. Moreover, evaluation data tended to be most often utilized in the 
following situations: when “red-flags” occur alerting staff of potential and/or existing problems in training courses; during 
curriculum revision and/or design phases; during presentations to upper-level management; immediately following a course’s 
conclusion in order to provide necessary feedback about that course; and, during corporate university budget revisions and 
modifications, as well as organization-wide budget cuts in order to illustrate the corporate university’s impact on the 
organization. 

The factors that were found to influence the utilization of evaluation data listed in Table 2 were very consistent 
throughout all of the sites. While the majority of the factors affected use in a positive sense, some were noted as having a 
negative impact on utilization. In order to increase evaluation use in a corporate university as well as an entire organization, 
these factors need to be taken into account throughout the entire evaluation process to ensure that those that are positively 
influential are promoted, while those that inhibit use are avoided. 

Conclusions and Recommendations 

Users of Evaluation Information. This investigator concludes that within the organizations and corporate 
universities examined in this study, a wide range of users exists. The assortment of users, in terms of their varied capacities 
and understanding and devotion toward evaluation use, has helped to communicate the importance of evaluation utilization to 
a large percentage of employees within these four organizations. As a result, in all of the sites, not only are training programs 
being evaluated using systematic techniques and methods, but the overall utilization of evaluation data for a variety of 
purposes has increased. 

Components of Evaluation Report Utilized. Throughout all four sites, because in many instances users’ preferred 
reporting formats were unknown, evaluation staff integrated multiple reporting techniques into their reports to ensure that all 
users were able to understand and utilize the assessment information. This investigator has concluded that in order to increase 
utilization of evaluation information in corporate universities and throughout their organizations, both quantitative and 
qualitative assessment data should be reported in a variety of formats including: executive summaries; a section of the report 
that uses descriptive statistics, charts, and graphs; and qualitative written descriptions of the findings that supplement the 
quantitative statistical breakdown of the data, in order to appeal to the different processing styles and preferred formats of all 
users. 



Uses of Evaluation Data. Given this study’s findings, it can be concluded that evaluation results are being used at 
the four sites for a variety of purposes ranging from internal corporate university training program improvements to the use of 
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data as a justification and accountability mechanism for the corporate university. While the nine purposes found in this study 
represent those that were most prevalent, a number of other less predominant purposes were also found at individual sites and 
tended to be unique to the specific corporate university and organization. 

Influential Factors Associated with Evaluation Utilization, In regard to the applicability of Cousins and 
Leithwood’s 1986 meta-analytic framework to the four sites, seven of the twelve factors from their framework were 
found to impact evaluation utilization in the corporate universities. Based on these findings, this investigator 
concludes that among the twelve factors identified by Cousins and Leithwood (1986), seven of them do not 
differentiate between the different contexts where program evaluation is occurring, in terms of affecting the use of 
assessment results. 

Six of the seven factors identified in this study that were found to influence training program evaluation use, are 
categorized under Cousins and Leithwood’s (1986) higher-order heading, evaluation implementation. The seventh factor, 
commitment and/or receptiveness to evaluation, falls under the second higher-order category in their model called 
decision/policy setting. Based on these results, this investigator has concluded that factors related to aspects of the actual 
evaluation, evaluator, and overall assessment process, as opposed to factors associated with policy setting characteristics, are 
more applicable and have greater influence on evaluation utilization in a corporate university training program setting. This 
may be due to the fact that more decisions regarding policy occur in the federal, public, and private contexts (social services, 
education, and healthcare) analyzed by Cousins and Leithwood (1986), than in the organizations containing corporate 
universities that were examined in this study. While factors that are related to aspects of the evaluation process are clearly 
most influential in this study, because setting policy did not appear as a top priority and purpose of the users of data in the four 
sites, it is not surprising that factors associated with Cousins and Leithwood's (1986) higher-order category of decision/policy 
settings were not found to be major factors affecting utilization. 

Evaluators 

Evaluators in corporate universities need to consider new potential users of evaluation information from 
both within the corporate university, as well as from the entire organization. These individuals may include other 
evaluators, trainers, instructional designers, student advisors, deans, and members of upper-level management. It is 
recommended that evaluators gain an understanding of the purposes for which the user intends to utilize the 
evaluation results and design processes and assessment tools/instruments to collect information that meets the user's 
needs. 

In order to increase evaluation utilization in their organizations, evaluators should implement a procedure that 
facilitates communication with users throughout the entire evaluation process (i.e., design, implementation, analysis, and 
reporting) in order to understand their needs, timeline, and preferred method of reporting. The evaluator should take proper 
action to match the reporting timeline to the needs of the user. By building a relationship with the various users of evaluation 
information and nurturing ongoing communication with them throughout the evaluation process, the evaluator's credibility 
may increase, the users’ commitment and/or receptiveness may rise, and the data that emerges from the evaluations may be 
more relevant to the users’ needs. 

When reporting evaluation results to internal corporate university staff, evaluators should include in-depth and 
detailed statistical breakdowns of the data according to evaluation item, and also supplemental, written explanations that 
describe the information. When reporting to the dean of a corporate university and to upper-level management, it is 
recommended that evaluators include statistical breakdowns by course, summary charts, graphs, and executive summaries. 

Managers in Corporate Universities 

Managers from within corporate universities should use this study’s findings to identify potential ways that 
they can utilize evaluation data in their roles. Specifically, managers should use evaluation data to demonstrate to 
upper-level management the corporate university's status/progress. Moreover, it is recommended that managers use 
evaluation information to assist in decisions regarding employee job placement. Managers should also use 
assessment data to market training courses and to increase visibility of the corporate university. 

Managers should request in-depth and detailed statistical breakdowns of the data according to evaluation item from 
evaluators. In addition, it is recommended that they ask evaluators for supplemental, written explanations that describe the 
evaluation findings. Deans of corporate universities should establish systems that encourage use among corporate university 
staff. 



Upper-Level Management. During the initiation, development, and refinement of an evaluation department 
within a corporate university, the findings from this study should be taken into account to ensure that training 
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program evaluation data are utilized to their fullest capacity. Specifically, it is recommended that managers who are 
interested in developing a corporate university establish an internal evaluation department. This may help to boost 
evaluator credibility throughout the organization, increase user commitment and understanding of evaluation, and 
promote ongoing communication between the evaluator and the user(s) through the entire evaluation process. 
Management should employ a knowledgeable and credible evaluation staff that is well-versed in psychometrics and 
is able to communicate evaluation results through a variety of formats including statistical control charts, graphs, 
and executive summaries. The evaluation staff should have both quantitative and qualitative analysis and reporting 
skills. Furthermore, it is recommended that the corporate university staff be trained in how to interpret and use the 
evaluative information. 

Researchers 

With the growing number of corporate universities in organizations throughout the United States (Meister, 1 999), 
additional research that focuses on evaluation use in these entities should be conducted. As new corporate universities 
emerge in a variety of industries, additional users of data, purposes of use, and factors that impact utilization may arise. These 
need to be explored in future studies. 

The factors found to impact use in a corporate university setting should be tested for their applicability in corporate 
training programs that exist outside of the corporate university context. The similarities in terms of curriculums that may exist 
between corporate training programs and those found in corporate universities would provide a basis upon which to study 
factors affecting evaluation use. By determining influential factors that impact data utilization in corporate training programs, 
it may encourage evaluation use to increase within this context. 

Further research focused on comparing evaluation use in a variety of contexts should be conducted in order to 
determine common purposes of utilization and factors influencing use that are consistent across settings where program 
evaluation is occurring. The results of such a comparative analysis would add to the literature on evaluation use. 
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Organizations that wish to engage in evaluation as part of their approach to organizational 
learning should consider determining how ready the organization is for learning from evaluation 
processes and findings. This paper reports the outcomes of implementing a diagnostic instrument 
with eight U.S. organizations. The findings indicate that the instrument is able to identify areas of 
strength on which to build evaluation efforts, and areas in need of development for creating and 
sustaining learning from evaluation work. 

Keywords: Evaluation, Organizational Learning, Survey Research 

Within the last few years, organizations have struggled to find ways to adapt, survive, and even thrive in today’s 
dynamic, unpredictable, and constantly changing marketplace and society: One popular approach to addressing 
organizational change and development has been to implement systems and processes that support organizational 
learning (see for example, Argyris & Schon, 1978, 1996; DiBella & Nevis, 1997; Dixon, 1994; Marquardt, 1996; 
Senge, 1990; Watkins & Marsick, 1993). The goal of implementing such systems and processes has been to become 
a “learning organization.” While many definitions of organizational learning abound in the literature, Torres, 
Preskill and Piontek (1996) define it as, “a continuous process of organizational growth and improvement that: (a) is 
integrated with work activities, (b) invokes the alignment of values, attitudes, and perceptions among organizational 
members, and (c) uses information or feedback about both processes and outcomes to make changes” (p. 2). In this 
sense, organizational learning represents the organization’s commitment to using all of its members’ capabilities. 
Ultimately, organizational learning is about creating continuous processes and mechanisms for learning about how 
to do things better. 

Another effort that is gaining attention within the learning and performance arena is evaluation (Bassi, 
Benson & Cheney, 1996; Brown & Seidner, 1998; Industry Report, 1996; Raelin, 2000; Sadler-Smith, Down & 
Field, 1999). As organizations seek to provide timely, relevant, cost efficient and effective programs, services, and 
products, they are asking for evaluative information to guide their decision-making practices. In spite of a desire to 
conduct more evaluations, however, employees are often uncertain about how to conduct rigorous, useful 
evaluations or how to find the resources and time to do them well. Few employees have any academic background 
in evaluation, or much practical experience conducting systematic evaluations. Further complicating the situation is 
the fact that even those who understand the value of evaluation worry that their organizations are not “ready” or 
really willing to evaluate; that the organization’s culture and leadership don’t appear to support evaluation, and that 
there are few processes, systems or mechanisms for conducting evaluation. Nevertheless, there remains a growing 
need and concern in how to effectively evaluate learning and performance efforts. 

Given this increasing interest in conducting evaluations and becoming a learning organization, some means of 
diagnosing an organization’s readiness to learn from evaluative inquiry is needed. It would clearly be a waste of 
time and resources to engage in evaluation work if an organization is not ready or interested in using the 
evaluation’s results or learning from evaluation practice. Therefore, the use of a diagnostic instrument can yield 
information helpful for preparing and guiding the organization in their efforts to evaluate various learning and 
performance programs, processes, products, systems, and services. 

Evaluation and Learning 

Preskill and Torres (1999) envision evaluative inquiry as: 

An ongoing process for investigating and understanding critical organizational issues. It is an 

Copyright © 2001 Hallie Preskill, Brenda Martinez-Papponi, Rosalie T. Torres 
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approach that is fully integrated with an organization’s work practices, and as such engenders 
(a)organization members’ interest and ability in exploring critical issues using evaluation logic, (b) 
organization members’ involvement in evaluative processes, and (c) the personal and professional 
growth of individuals within the organization, (p. 1-2). 

They propose that evaluative inquiry is most successful when four learning processes are used throughout an 
evaluation. These include: (1) Dialogue, (2) Reflection, (3) Asking Questions, and (4) Identifying and Clarifying 
Values, Beliefs, Assumptions, and Knowledge. When an evaluation is facilitated using these processes, greater 
insights and understandings about the evaluand (that which is being evaluated) are developed. These learning 
processes also support a collaborative and participatory approach to evaluation throughout the (1) focusing the 
evaluation, (2) carrying out the evaluation, and (3) applying learning phases. 

According to Preskill and Torres (1999), the organization’s infrastructure also directly affects the 
effectiveness of any evaluative inquiry effort. They identify four elements of this infrastructure: (1) Culture, (2) 
Leadership, (3) Communication, and (4) Systems and Structures. These elements form the foundation on which 
evaluative inquiry efforts can be undertaken and sustained. That is, these elements will facilitate or inhibit 
individual, team, and/or organizational learning from evaluative inquiry to varying degrees, depending on how they 
function within the organization. 

Administering a diagnostic instrument such as the one discussed in this paper, prior to conducting an 
evaluation, can help organization members understand the degree to which the evaluation will be successful by 
considering the extent to which different organizational elements support or inhibit learning from evaluation. For 
example, an organization may learn that support for evaluation will be limited, or that certain elements of its 
infrastructure do not support inquiry and learning in general. This organization would then know that attention to a 
wider array of its practices and systems would be needed for evaluative inquiry to be maximally successful. And 
when an external evaluator is involved, sharing this information will enable her to know beforehand, what 
challenges to expect with regard to the organization’s ability to learn from the evaluation, or use its results. Being 
aware of this enables the evaluator to address these obstacles to use so that the evaluation can be as effective as 
possible. 

Research Purpose and Guiding Questions 

The purpose of the research was to further establish the validity of the Readiness for Organizational Learning and 
Evaluation (ROLE) instrument and to understand the status of evaluation and learning readiness in eight pilot 
organizations. Specifically, this study sought to answer the following questions: 

1. What are the strongest and weakest dimensions across the eight pilot organizations (Culture, Leadership, 
Communication of Information, Systems and Structures, Teams, and Evaluation)? 

2. Do respondents’ perceptions differ based on their position in, and longevity with, the organization? 

3. How do the dimensions’ mean scores vary across the eight organizations? 

4. What challenges might these organizations face when engaging in evaluation work? 

Methods 

Data were collected using the newly designed Readiness for Organizational Learning and Evaluation (ROLE) 
instrument. The instrument was developed based on a) a review of the organizational learning and program 
evaluation literatures, b) a review of existing organizational learning and readiness for evaluation instruments, and c) 
interviews conducted with several organizations that are attempting to become “learning organizations.” The items 
were classified into dimensions based on the framework described in Evaluative Inquiry for Learning in 
Organizations (Preskill & Torres, 1999). These dimensions include: (a) Culture - 26 items, (b) Leadership - 12 
items, (c) Communication of Information - 8 items, (d) Systems and Structures - 18 items, (e) Teams - 1 1 items, and 
(f) Evaluation - 9 items. With three additional demographic items, the total number of items on the instrument is 87. 

Sample 



To establish the instrument’s construct validity, eight organizations were invited to administer the 
instrument with their employees. Each of the organizations had already begun making efforts to become a learning 
organization and saw the instrument as a means to understand how well they were doing, and their members’ 
interest in integrating evaluation into their work practices. 
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Because we hope the instrument will be useful in a wide range of organizations, we sought to involve a 
diverse group of organizations in the study. The participating organizations included a: hospital, non-profit 
community organization, elementary school, car dealership, Fortune 50 technology company, newly established 
community college, department within a university’s medical school, and county administrative office. These 
organizations were located in the Northwest, Southwest, Northeast, and Rocky Mountain regions of the U.S. Each 
organization was invited to participate at the organization or department/unit level. In five of the organizations, 
respondents rated the items thinking about their organization overall, while three responded in terms of their 
department or unit. A total of 232 useable instruments were returned from the eight organizations. For their 
participation, each organization received a 1) summary report that included descriptive statistics for each item, 2) 
aggregate scores for each dimension, 3) a narrative summary of the results, and 4) an offer for two hours of free 
consultation regarding their organization’s results. 

While details on establishing the instrument’s construct validity can be found elsewhere (Preskill, Torres, 
& Martinez- Papponi, 1999), the instrument was found to be internally consistent (Cronbach’s alpha .97 across the 
78 Likert scale items, and a range of .83 to .94 across the six dimensions). Exploratory factor analyses suggested the 
following modifications to the original instrument: (1) elimination of four items with low communality, (2) creation 
of three subscales within the culture dimension; (3) creation of three subscales within the systems and structures 
dimension; and (4) creation of two subscales within the communication dimension. 

The findings reported in this paper reflect the 232 responses to the original instrument before final changes 
were made based on the factor analyses. In spite of reorganizing some of the items and the elimination of four 
items, none of the items were reworded; thus, we are confident the results reported here are valid and worthy of 
discussion. 

Data Analysis 

For each question, participants were asked to indicate on 5-point Likert scale (ranging from 1 = Strongly Disagree to 
5 = Strongly Agree) the extent to which they agreed with each statement. Data were analyzed using the SPSS 
(version 9.0) statistical package. Descriptive statistics (means and standard deviations) were calculated for each 
instrument item, as well as for each of the six dimensions (e.g.. Culture, Leadership). When a design consists of 
more than two dependent variables, it is necessary to determine which ones differ significantly from each other. 
Therefore,. a MANOVA using the mean scores for each dimension as the dependent variable was conducted. Since 
missing data constituted less than 1% of the responses, the series means for each dimension were used for the 
missing data in that dimension. 

The general linear repeated-measures model provides analysis of variance when the same measurement is 
made several times on each case. In this situation, we treated the dimension as the repeated measure (within 
subjects). We also specified a between-subjects factor (organization) and therefore, the population was divided into 
groups. Using the repeated measures model, we were able to study the effects of both the between-subjects factors 
and the within-subjects factors. Further, we were able to investigate interactions between factors as well as the 
effects of individual factors. Series means for each dimension were again used for the analysis for missing data in 
that dimension. 

Findings 

The findings from this study are reported for each research question. Before discussing the results, however, it is 
important to remind the reader that the participating pilot organizations agreed to become involved because they 
were interested in enhancing their organization’s evaluation and learning capabilities. In some cases, they had 
already implemented several organizational learning processes and activities. Therefore, most of these organizations 
were predisposed to the basic concepts of team and organizational learning, which might have skewed the overall 
results in a more positive direction. We do not intend to generalize these results to other organizations. Rather, we 
believe the results provide additional insights into the instrument’s reliability and validity, and some trends in these 
particular organizations. 

1. What are the strongest/weakest dimensions across the eight pilot organizations? 

As can be seen in Table 1, the highest rated dimension across the eight organizations (N=232) is Teams, 
while the lowest rated is Systems and Structures. Only those respondents who said they work in teams (n= 193/84%) 
completed the questions in the Teams dimension. These items focused on how well team members: (a) deal with 
conflict, (b) are open and honest with one another, (c) facilitate meetings, (d) accomplish their work, and (e) are 
encouraged to share their learning with others. Interestingly, of these 193 individuals, only 101 or 44% said they 
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have received training on how to work as a team member. Nevertheless, it appears that overall, respondents believe 
that their teams are working and are an effective means for completing various work-related tasks. 

Table 1 . Means and Standard Deviations for Each of the Six Dimensions 



Dimension 


Mean 


Standard Deviation 


Teams (n = 185) 


3.7561 


.7222 


Leadership (n = 232) 


3.6219 


.7474 


Culture (n = 232) 


3.5326 


.6269 


Evaluation (n = 232) 


3.5268 


.5673 


Communication of Information (n = 232) 


3.4339 


.6490 


Systems & Structures (n = 232) 


3.2780 


.5683 



The Systems and Structures dimension, rated lowest by respondents across the eight organizations included 
items that focused on (a) the accessibility and openness of the work environment, (b) the organizations’ rewards and 
recognition systems and practices, and (c) the relationship of employees’ work to the organizations’ goals. A profile 
analysis for the eight organizations as shown in Figure 1 is another way to view the highest and lowest dimensions 
for each of the organizations. As can be seen in this figure, the Teams dimension was consistently higher across 
organizations, just as the Systems and Structures dimension was consistently lower across the organizations. 

Figure 1 . Profile Analysis of Eight Participating Organizations 



Profile Analysis 



—♦—Culture 

— Leadership 

—^—Systems & Structures 

""^Communication of 
Information 

-"^Evaluation 

—•—Teams 



1 2 3 4 5 6 7 8 

Organizations 




2. Do respondents’ perceptions differbased on their position in, and longevity with, the organization? 

These analyses were conducted using the MANOVA tests and did not include responses to items in the Teams 
dimension. Since this was the only dimension that did not require responses from all participants, it was not 
included as a variable in the remaining analyses. 

We used a repeated measures multivariate analyses of variance (MANOVA) to examine whether being in a 
supervisory or management position affected how respondents rated each item. Using a 5 (dimension) x 2 (position) 
design, we combined “First-line Supervisor, Middle Manager, and Senior Manager” into one group (n=44), and the 
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rest of the respondents’ into another group (n=164). We questioned whether those in supervisory roles would have a 
more positive view of how ready the organization was for learning and evaluation. The results of the MANOVA 
revealed a multivariate effect, Wilks’ Lambda = .95, F (5, 202) =. 2.18, p < .057. This finding shows that managers 
did rate the dimensions differently than those in non-managerial positions. When we examine the individual means 
and standard deviations (See Table 2), we see that the managers rated the dimensions higher (more positively) as we 
thought they might. In other words, they were more likely to agree that their organizations reflect a learning and 
evaluation environment. 

Table 2. Means and Standard Deviations for Each Dimension Based on Position Within the Organization 



Dimension 


Manager (1) or 
Non-Manager (2) 


Mean 


Standard Deviation 


Culture 


1 


3.5323 


.5122 




2 


3.5055 


.6504 


Leadership 


I 


3.7875 


.5756 




2 


3.5770 


.7795 


Systems & Structures 


1 


3.2521 


.5059 




2 


3.2455 


.5772 


Communication of Information 


1 


3.3636 


.5686 




2 


3.4350 


.6821 


Evaluation 


I 


3.6204 


.5558 




2 


3.4755 


.5537 



To determine if the length of one’s time with the organization affects the overall dimension ratings, a 5 
(dimension, excluding Teams) x 3 (longevity) MANOVA was performed. Participants were divided into three 
groups based on their time with the organization; less than a one-year (n=49), one to three years (n=76); and four 
years or more (n=87). Results of the MANOVA were not significant (p = .23), indicating that the dimension 
variables did not differ significantly between the three groups. Individual means and standard deviations for the 
dimensions are reported in Table 3. 

Table 3. Means and Standard Deviations for Each Dimension Based on Longevity with the Organization 



Dimension 


Longevity 

1= less than one year; 

2= 1-3 years; 

3 = 4 or more vears. 


Mean 


Standard 

Deviation 


Culture 


1 


3.6950 


.5808 




2 


3.4954 


.6770 




3 


3.4356 


.5876 


Leadership 


I 


3.8348 


.7377 




2 


3.6010 


.7650 




3 


3.5387 


.7200 


Systems & Structures 


I 


3.4142 


.5653 




2 


3.2066 


.6088 




3 


3.2014 


.5188 


Communication of Information 


1 


3.4808 


.6680 




2 


3.4462 


.6873 




3 


3.3649 


.6395 


Evaluation 


I 


3.5581 


.6675 




2 


3.5829 


.5543 




3 


3.4160 


.4919 



3. How do the dimensions’ mean scores vary across the eight organizations? 

A repeated-measures ANOVA (using deviation contrasts, which compares the mean of each level to the 
grand mean) test was preformed in order to examine if the dimensions (excluding Teams), differed between each 
other, and if the differences in dimensions differed across the organizations. The repeated-measures ANOVA 
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confirmed that participants do vary across dimensions, Wilks’ Lambda = .70, F (4, 221)= 23.61, p < .000. The 
analyses also showed that there were significant differences across the eight organizations as they rated the five 
dimensions, Wilks’ Lambda = .63, F (28, 798) = 3.96, p < .000. These results further validate our earlier findings 
that the dimensions differ among themselves — that the items are discreet from one another across the dimensions. 

4. What challenges might these organizations face when engaging in evaluation work? 

To better understand specific issues for the organizations within each dimension, the instrument items were 
ordered by the size of their means (in ascending and descending order). As can be seen in Table 4, of the top ten 
item means across all of the organizations, four are in the Teams dimension, three are in the Culture dimension, two 
are in Leadership, and one is in the Systems and Structures dimension. These findings imply that respondents 
believe their teams are accomplishing work in ways that support individual and team learning. In addition, there is 
some reason to think that in most of the participating organizations, the organization’s culture supports the sharing 
of knowledge, asking questions, and using each other as resources. 

Table 4. Ten Highest Ranked Item Variables According to Means 



Instrument Item 


N 


Mean 


#75 ‘Teams are an effective way to meet an organization’s goals.” (Teams) 


190 


4.15 


#2 “Employees ask each other for information about work issues and activities.” (Culture) 


232 


4.13 


#36 “Managers and supervisors support the sharing of knowledge and skills among employees.” 
fLeadershin) 


230 


3.95 


#1 “Emplovees respect each other’s perspectives and opinions.” (Culture) 


230 


3.91 


#8 “Asking questions and raising issues about work is encouraged.” (Culture) 


232 


3.90 


#79 “Evaluation would help us provide better programs, processes, programs and services.” 
(Evaluation) 


228 


3.89 


#72 “Team meetings strive to include everyone’s opinion.” (Teams) 


191 


3.87 


#76 “Integrating evaluation activities into our work would enhance the quality of decision-making.” 
(Evaluation) 


228 


3.86 


#74 “Teams accomplish work they are charged to do.” (Teams) 


192 


3.86 


#73 “Teams are encouraged to learn from another and to share their learning with others” (Teams) 


190 


3.81 


#35 “ Managers and supervisors believe that our success depends upon learning from daily practices.” 
(Leadership) 


229 


3.81 


#53 “Emplovees meet work deadlines.” (Systems & Structures) 


231 


3.81 



Table 5 presents the items with the lowest mean ratings across the eight organizations. As can be seen, six 
of the ten lowest rated items are in the Systems and Structures dimension, with two in the Culture, one in the 
Evaluation, and one in the Communication of Information dimension. 

The overall picture for these pilot organizations suggest that (a) employees have more work than they can 
accomplish in timely ways, (b) find little support for trying something new, and (c) don’t believe the reward and 
recognition system sufficiently supports their learning or experimenting with new ideas. In addition, there appears 
to be few mechanisms for evaluation or for documenting the success of past change efforts. While the aggregated 
data do show some interesting trends across the eight organizations, it is important to note that each individual 
organization had their own highest and lowest rated items within each dimensions and thus have different strengths 
and challenges as they attempt to learn from evaluation and their practice. 

Discussion 

The results of this research further confirm that the ROLE instrument effectively measures the constructs of 
organizational learning and evaluation. Not only did respondents vary in how they answered each item, but they 
also varied in how they rated each dimension overall across the eight dimensions. In other words, respondents were 
able to determine the discreet nature of each instrument item. These findings suggest that the instrument may indeed 
be useful to organizations that are interested in understanding where their strengths and weaknesses are relative to 
integrating evaluation into their organizational learning practices. 
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Table 5. Lowest Ranked Item Variables According to Means 



Instrument Item 


N 


Mean 


#55 “Employees are able to get their work done in a timely way.” (Systems & Structures) 


230 


2.81 


#49 ’’Work commitments do not exceed employees’ capacity to fulfill them.” 
(Systems & Structures) 


231 


2.87 


#84 “There are evaluation processes in place that enable employees to review how well changes we 
make are working.” (Evaluation) 


224 


2.94 


#61 “There are adequate records of past change efforts and what happened as a result.” 
(Communication of Information) 


227 


2.97 


#40 “There is little bureaucratic red tape when trying to do something new or different.” (Systems & 
Structures) 


229 


3.02 


#20 “Employees generally view problems or issues as opportunities to learn.” (Culture) 


232 


3.04 


#50 “Employees are recognized or rewarded for helping each other learn.” (Systems & Structures) 


230 


3.05 


#5 1 “Employees are recognized or rewarded for experimenting with new ideas.” (Systems & 
Structures) 


230 


3.06 


#46 ’’The current reward or appraisal system recognizes, in some way, team learning and 
performance.” (Systems & Structures) 


230 


3.08 


#6 “Employees are more concerned about how their work contributes to the success of the 
organization than they are about their individual success.” (Culture) 


229 


3.18 



The finding that there were significant differences between those in a supervisory or management role was 
not particularly surprising, since we initially thought that managers might be more positive about how well their 
organizations or departments embody organizational learning principles. It often seems that those most closely tied 
to the daily operations of the organization — those that are on the front line, have different perceptions about how the 
organization supports their work. At the same time, we need to be cautious in interpreting these results. In 
categorizing the two groups, we combined respondents who checked that they were a “First-line supervisor,” 
“Middle manager,” or “Senior manager.” The second group comprised those who checked “Administrative, 
Production, Sales, Non-Managerial Professional, Technical, Customer Service, or Other.” We debated whether 
those in the “Administrative” category might also have supervisory responsibilities, and finally decided to place 
them in the non-managerial group. Thus, we cannot be certain that those in “Production,” for example, might not be 
supervisors or managers in that specific area. In addition, the sample size for each group was uneven (n=44 for 
managers, and n=164 for non-mangers). To further understand whether real differences exist for these two groups, 
additional research using a more discreet classification scheme is warranted. 

The length of time one has worked for the organization, however, does not appear to affect how 
respondents answered each item. We were somewhat surprised at this finding since we thought that newer 
employees would perceive the organization’s support for organizational learning and evaluation practices differently 
than more veteran employees. In some ways, this is good news, since it may be one less variable that needs to be 
considered if implementing the instrument within an organization. However, once again, further research should 
determine if this finding holds up for a larger sample. 

Of particular interest is the result of the highest and lowest rated dimensions. The fact that the lowest rated 
items clustered in the Systems and Structures dimension, suggests some serious obstacles to employee learning and 
evaluation within organizations. The lack of time to do their job, being faced with a bureaucracy that impedes 
cxpcriiTivntat'iGn, the lack of a rev/ard and rccw^nition system thut SupiJ^rts anu me iaeiv any liiean^ lw 

track progress on various initiatives, are all areas that will inhibit organizational learning and evaluation work. 
While these results present some concern, we suspect that these findings are not unique to the eight organizations 
included in this study. Future research might explore how organizations have overcome some of these obstacles or 
test various organizational change efforts that seek to create more learning oriented systems and structures. At the 
same time, given the seeming impossibility of changing an organization’s systems and structures or culture, research 
might instead focus on how organizations can capitalize on their strengths to overcome deficiencies found in the 
other dimensions. 



Conclusions 

We believe evaluation is a useful approach to helping organization members answer pressing questions that 
challenge their success and future. In so doing, it can provide critical information for making decisions that result in 
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increased individual, team, and organizational learning. Yet, it is often difficult to know where to start within an 
organization that is trying to integrate evaluative inquiry into its daily work practices. The Readiness for 
Or ganizational Learning and Evaluation (RQLE^ instrument is one tool we believe can help organization members 
and learning and performance practitioners in particular, understand an organization’s readiness to engage in, and 
learn, from evaluative forms of inquiry. 
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Three Perspectives of Training Evaluation Based on Organizational Needs 
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This article suggests that matching the intents of evaluation stakeholders and the evaluation 
emphasis has not been addressed in HRD, It is important for HRD professionals to better match 
the intents of evaluation, the intended users, and the aspects of evaluative process and 
information. Three perspectives of stakeholders on evaluation of training were developed as a 
means for ensuring the match between the intents and activities of evaluation. 

Key words: Evaluation, Needs Assessment, Organizational Research 

As the business environment becomes more competitive due to global influences, technological advancement, and 
other factors, organizations are more interested in developing high-performance workplace (Faerman & Ban, 1993). 
Thus organizations place more emphasis on developing workforce’s skills, and knowledge through continuous 
learning to remain competitive. In return for this employer-supported training, organizations anticipate increased job 
effectiveness and performance. Higher tangible and intangible impacts are expected from organizations that spent 
over 58 billion dollars on employee training and education in both 1995 and 1996 (Bassi, Benson, & Cheney, 1996). 

Organizations are demanding more accountability from providers of HRD programs for their employees 
(Robinson & Robinson, 1995; Swanson & Holton, 1999). Thus HRD professionals in organizations are increasingly 
being asked to show how their efforts add value to their organizations. In many organizations, training evaluation is 
thought to be the most appropriate method of demonstrating this value (Preskill, 1997). While the importance of 
training evaluation has been emphasized by both scholars and practitioners (Brinkerhoff, 1987; Rouiller & 
Goldstein, 1993; Kirkpatrick, 1994), there have been few systematic evaluations to demonstrate the effectiveness of 
training programs by either organizational managers or trainers (Dionne, 1996; Faerman & Ban, 1993). 

With the advent of Kirkpatrick’s four level evaluation model, many models have been suggested to help 
managers and trainers measure and evaluate the efficiency and effectiveness of training programs (Baldwin & Ford, 
1989; Brinkerhoff, 1987; Holton, 1996; Kaufman & Keller, 1994; Noe & Schmitt, 1986; Swanson & Holton, 1999). 
Unfortunately, evaluation models are seldom used wisely or used to answer the most vital questions that face every 
organization. Preskill (1997) argues that most of these models were not grounded in a philosophy or theory of 
evaluation and position evaluation as a periodic event that is the sum of a set of technical skills and activities focuses 
on a narrow set of variables using limited numbers of designs and methods. Holton (1996) suggests that at best most 
models are simple taxonomies or classification schemes because they do not fully address all constructs underlying 
the phenomena of interest. As a result evaluation of training has focused on a narrow set of questions using tools and 
methods and then has failed to show HRD’s contribution to organizations. 

HRD scholars and practitioners agree that training programs influence and are influenced by elements of a 
larger organizational system (Brinkerhoff, 1997; Rummler & Brache, 1995; Swanson, 1994). They advocate that 
training program be thought of in terms of improving performance in its broader systemic context. After all, the 
evaluation of training must be treated as the study of a part of an organizational subsystem that must be understood 
within an organization context (Dionne, 1996; Tannenbaum & Woods, 1992) where the roles, responsibilities, and 
tasks of evaluation vary with both time and circumstances. 

The objectives of training programs reflect numerous goals ranging from trainee progress or training 
program improvement to organizational goals. Every stakeholder of training program in organizations intends to 
judge training activities against his or her own criteria, and the information they seek is not used for the same 
purposes. There is a whole set of values and attitudes that belong to the trainees, the trainers, and the HRD 
managers, the evaluators, and the decision-makers in the organization. It would be naive to suggest that these values 
and attitudes do not affect many of the decisions involving both the evaluation and the resulting data interpretations. 
Thus it is necessary to shift attention from the methods or object of evaluation to the intended users of evaluative 
processes and information, and their intended uses (Patton, 1997). Accordingly, organizations should evaluate its 
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training programs in terms of achieving the goals of evaluation while the goals should derive from the needs of the 
intended stakeholders of evaluative processes and information. 

The failure of past evaluations has to do with the fact that most of organizations has selected the questions 
and methods of training evaluation or adhered to a particular perspective of training evaluation without having clear 
goals of evaluation. A useful evaluation should be responsive to the questions and needs that organizations and their 
stakeholders must face. In this regard, it will be important that evaluators help stakeholders of training programs 
identify their evaluation needs, determine what information and processes would best address those needs, and 
generate findings what would yield the type of information needed by the intended stakeholders. 

The purpose of this study is not to present another evaluation model, but to suggest a training evaluation 
framework that organizations can consult when evaluating their training programs. The framework proposed here is 
flexible enough to accommodate most any evaluation model and may add components favorable to structural 
integration with the system of training activities to be evaluated. First, this study describes three perspectives of 
training evaluation, including their definition, the issues or questions of evaluation to be addressed, and 
representative evaluation models. Second, the study concludes with ideas on how the proposed framework may 
contribute to HRD practice and research. 

Theoretical and Research Questions 

The framework of evaluation proposed here intends to address the following questions: 

1. What are major perspectives of training evaluation on which HRD practitioners and researchers are based 

on? 

2. What evaluation issues do each perspective of training evaluation address? 

3. What are the implications of three perspectives of training evaluation on HRD practice and research? 

Three Perspectives of Training Evaluation 

Despite the fact that the approaches and models of evaluation adopted by HRD practitioners and researchers are 
almost as varied as their proponents, HRD practitioners and researchers have only explored the issues of the models 
or methods of training evaluation. They have not paid full attention to the intents of evaluation, particularly on the 
part of HRD professionals or their organizations. Patton (1997) argues that evaluation without an intention to use the 
findings should be meaningless at all. For an evaluation to be effective and efficient, there should be a plan for the 
intentional use of findings. Otherwise, investing the resources in an evaluation may be questionable. There seems 
quite a high degree of consistency about what is considered to be the primary intents of evaluation underlying 
evaluation activities. In Table 1, those intents of evaluation are described in terms of three perspectives of 
stakeholders in organizations on training evaluation. They are called as judgment-oriented, improvement-oriented, 
and learning-oriented evaluation. 

Judgment Perspective of Training Evaluation 

Known as a kind of instrumental use (Shadish, Cook, & Leviton, 1991), the judgment perspective of evaluation 
refers to the assessment of merit and worth of training program, thus making direct decision about changing training 
program based on evaluation results. Judgment perspective of training evaluation aims to determine the overall 
worth of training through developing a warranted judgment of the important characteristics and the value of the 
training. The judgment perspective of training evaluation helps HRD professionals to: 1) judge the overall 
effectiveness of the training program; 2) make decisions about continuing or terminating the training program; 3) 
make sure that the training program be implemented and effective as planned; 4) allocate new budget outlays for the 
training program; and 5) accomplish fundamental change efforts in how organizations achieve their goals. In short, 
this perspective of evaluation demonstrates conclusively that something has happened as a result of training 
activities. This will be linked to judgments about the value of the training program. 

The evaluation models or approaches based on judgmental perspective are labeled the outcomes approach 
because the emphasis is on demonstrating that the appropriate outcomes occur (Holton & Hannigan, 2000). Since 
Kirkpatrick’s four-level model of reaction, learning, behavior, and results (Kirkpatrick, 1994), a variety of HRD 
scholars have offered elaborations, updates, and variations in an attempt to improve the taxonomy. Kaufman and 
Keller (1994) have proposed the addition of societal impact as a fifth level and Brinkerhoff (1987) offered a six- 
stage model that added two more level of formative evaluative process to Kirkpatrick’s model. Through the results 
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Table 1. Three Perspectives of the Evaluation of a Training Program 





assessment system, Swanson and Holton (1999) have tried to provide HRD practitioners with the systematic 
evaluation process for assessing learning, performance, and perceptions. With an addition of return on investment to 
the taxonomy, Phillips (1997) and Jacobs, Jones, Neil (1992) suggest that the financial benefits of training programs 
be forecast because organizational results are more likely to be achieved if the benefits are calculated and known to 
stakeholders involved in the training. 

Sometimes even if it is possible to overcome the technical difficulties in measuring outcomes and changes 
resulting from training activities, it has still to be decided against whose criteria of value such changes might be 
assessed. Patton (1997) argues that specifying the criteria for judgment is central and critical in judgmental 
evaluations. Scriven (1991) also emphasizes the importance of establishing the evaluation criteria when he presents 
the logic of valuing rules, which are 1) selecting criteria of merit; 2) setting standards of performance; 3) measuring 
performance; and 4) synthesize results into a judgment of value. Alliger, Tannenbaum, Bennett, Traver, and 
Shortland (1997) also insist that for any training evaluation to be valuable, training criteria must be meaningful to 
decision makers as well as psychometrically sound, and must be able to be collected within typical organizational 
constraints. 

Notions of what are good and desirable outcomes for trainees are not universal. They may be presented as 
if they are self-evident and objectively determined, but in fact they are articulated primarily to serve the interests and 
expectations of stakeholders within organizations. Stakeholders are defined as those who have an interest in a 
training event, and thus may include trainees, trainers, HRD managers, and upper-level managers. In terms of 
judgmental evaluation, evaluators should identify all stakeholders involved with the training and attempt to identify 
what criteria they would use in assessing the value of the training. Any evaluation information can then be collected 
with these multiple criteria in mind. 

Improvement Perspective of Training Evaluation 

The second perspective of training evaluation is improvement-oriented evaluation. This training evaluation refers to 
the efforts to provide timely feedback designed to modify and enhance training program operations. That is, a more 
casual and timely assessment of merit and worth is conducted, and the results are reported to training program 
managers and staff who can then use the feedback to establish the need for training program modifications. It also 
focuses on diagnosing the parts of he training program that are causing observed problems, considering alternative 
approaches to these parts of the training program, and deciding which change to pursue. The improvement 
perspective of training evaluation helps HRD professionals to: 1) improve the implementation of training program; 
2) solve unanticipated problems; and 3) make sure that trainees are progressing toward desired outcomes. In effect, 
the improvement-oriented evaluation is to provide information about the training methods that are most effective, 
offer guidance for improving the training operations, and identify difficulties and weaknesses within the processes. 
Along with the judgment-oriented evaluation, the improvement-oriented evaluation is very important because the 
evaluation is linked to cost-effectiveness that can lead to positive business results (Rohrer-Murphy, Moller, & 
Benscoter, 1997). In general, the followings can be considered as major parts of improvement-oriented evaluation. 

First, the improvement-oriented evaluation will concentrate on the performance of any critical training 
functions; on those aspects of the training with high visibility and importance, and on any areas where there may be 
indication of trainee dissent, conflict, or uncertainty, including trainer’s ones. This evaluation reviews the training 
operations in two different but interconnected ways. On the one hand, it focuses on the efficiency of training 
operations (Kaufman & Keller, 1994). It should be reviewed regarding how well resources (financial, materials, 
personnel, and so forth) are being used in the training program. It was common and necessary to report on whether 
the training program has met budgetary parameter and delivery schedules. On the other hand, it also examines how 
well trainees are reacting to the training activity and how well trainees progress toward expected outcomes. Trainee 
discontent or dissatisfaction or misleading can be a serious problem. Beyond judging trainees’ satisfaction to 
training program, the sources of the problems should be identified and corrections applied through evaluating 
training activity. 

Second, while training quality can be assessed and improved on the basis of the quality of training methods 
and materials or direct learning results, it is ultimately judged on its contribution to organizational goals. As 
Brinkerhoff and Montesino (1995) put it, a training that yields some effects but that was not needed is not efficient 
for the organization. For sure, identifying organizational needs and deriving the training goals from them must be an 
activity that needs to take place in the beginning stage of the design process (Goldstein, 1993). However, 
organizational needs may change in the course of time, so evaluators should constantly be aware of the 
organizational needs during training delivery. Those who emphasize the improvement-oriented evaluations will 
continue to examine if training delivery meets organizational needs over time. 
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Third, according to extensive research on transfer of training (Baldwin & Ford, 1988; Holton, 1996; Noe & 
Schmitt, 1986), training characteristics influence the transfer of training into the job, such as motivation, job- 
involvement, self-efficacy in successful completion of the training program, and ability to learn. The individual 
characteristics of trainees should be identified in the evaluation process because they determine the degree of 
successful learning and transfer. Furthermore, as the training should have impact on an organizational level, it is 
regarded as important to yield information about work related problems and needs as about personal ones. Trainees 
are less likely to apply the new skills if managers do not show that they value training. As a result, the training effect 
will be minimal. HRD practitioners are responsible for undertaking activities during the total training process to 
keep management and supervisors involved and should ask their help in implementing a successful training program 
(Holton, Bates, Seyler, & Carvalho, 1997). Evaluators should assess the degree of the management and supervisors’ 
involvement and commitment through the total training process. In terms of the training design, HRD researchers 
(Baldwin, Magjuka, & Loher, 1991) assert that trainees who had a choice of training content had greater motivation 
to learn. Noe and Schmitt (1986) also indicate that the degree to which a trainee is involved in the needs assessment 
process and given choices about training will influence to learn. Goldstein (1993) argues that trainees who are taught 
the manner in which to apply their knowledge on the job are likely to successfully transfer the learning to the job. 
Evaluators should assess the degree to which the trainees are involved in the training design or selection and the 
training content reflect the job context. 

In effect, the improvement-oriented evaluation extends its scope from traditionally focusing on training 
material, methods, and delivery schedule to continuous examination of organizational needs, assessment of training 
characteristics and work environment, and effective training design. If training will not yield to its desired outcomes, 
improvement-oriented evaluators should search for its causes in order to provide concerned stakeholders with the 
information used for enhancing training program. The information from improvement evaluation will serve the 
needs of organizational stakeholders who require developing training efficiency and effectiveness. 

Learning Perspective of Training Evaluation 

The third perspective of training evaluation is learning-oriented evaluation. Learning-oriented evaluation reflects 
two aspects of evaluation utilization. One involves the conceptual use. Conceptual use occurs when an evaluation 
influences decision makers’ and stakeholders’ cognitive processing (thinking) about a present or future program. In 
terms of conceptual use, learning-oriented evaluation seeks to become aware of evaluation results, become aware of 
features of a program from an evaluation, forming attitudes about a particular training program because of an 
evaluation and knowledge about evaluation in general (Johnson, 1998; Patton, 1997). The other involves, what is so 
called, the process use. Process use occurs when behavioral and cognitive changes occur in persons involved in 
evaluations as a result of their participation (Patton, 1997; Preskill & Caracelli, 1997; Schulha & Cousins, 1997). It 
results from experiential learning and reflection. 

In terms of both conceptual and process use, learning-oriented evaluation represents a view that evaluation can be a 
catalyst for learning that has the potential to improve and transform individuals and organizations. Supporting the 
role of evaluation in this aspect. Preskill and Torres (1999) suggested new definition of evaluation as follows. 
“Evaluative inquiry is an ongoing process for investigating and understanding critical organizational issues. It is an 
approach to learning that is fully integrated with an organization’s work practices, and as such, it engenders (a) 
organization members’ interest and ability in exploring critical issues using evaluation logic, (b) organization 
members’ involvement in evaluative processes, and (c) the personal and professional growth of individuals within 
the organization (p.1-2)”. Thus training evaluation may have long term payoffs through improved skills, improved 
communication, improvement of decision making, increased use of evaluation procedures, changes in the 
organization and increased confidence in and sense of ownership of evaluation products. 

In effect, learning-oriented evaluation refers to efforts to understand the training program better, enlighten 
top management, staff, and other stakeholders, enhance communications among them, and facilitate sharing of 
perceptions through engaging in training evaluation. In this regard, learning-oriented evaluation will help HRD 
professionals to improve organizational capacity to set training policies, design and administer training program, to 
support and reinforce the training program, to increase engagement, self-determination, and ownership of all 
stakeholders, and to rethink the ways to design and connect various programs for improving employee performance. 

Therefore, learning-oriented evaluation must be viewed as a strategic organizational activity. Evaluation 
has traditionally been treated as an after-thought or as a post-course reaction survey at the end of a training session. 
There is little interest on the role evaluation can play in helping organizations achieve their goals, or how evaluation 
can facilitate the organization’s growth and development. Learning-oriented evaluation provides HRD practitioners 
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and their organizations with the insights on how evaluative information and activities may induce cognitive and 
behavioral changes within an organization context. 

Contribution to HRD Practice and Research 

Organizations around the world are doing more training and spending more money than ever before. Unless 
organizational leaders in training engage in far greater use of evaluation and other quality improvement efforts than 
they now pursue, they face serious threats to its survival. The challenge is to do more of a different order of 
evaluation, that is, proactive and directive evaluation that will lead HRD profession and practice to a new level of 
operation and impact. 

Evaluation of training should be considered as a study of organizational subsystem that is only understood 
within an organizational context where the roles, responsibilities, and tasks of evaluation vary with time and 
circumstances. In this regard, different stakeholders for training evaluation may desire different evaluation 
information and processes. To provide meaningful training evaluation, it is important for HRD professionals to be 
able to match the intents of evaluation, the intended users, and the aspects of evaluative process and information. 

The primary justification for evaluation of training is its usefulness to concerned stakeholders. For the 
evaluation to be utilized, HRD professionals should do top quality evaluations that are responsive to the concerns of 
those whose training programs they evaluate, and they should do their best to communicate results in multiple ways 
to ensure that stakeholders know the results and see their relevance for program action. So HRD professionals 
should spend a substantial amount of time and efforts in learning about the informational needs of those for whom 
the evaluation is being done. 

For the HRD practitioners, the proposed three perspectives of training evaluation offer a means for 
identifying the intents of organizational stakeholders whose training program is evaluated and so preparing what 
questions must be answered by the evaluation and what methods must be employed to evaluate. With the proposed 
framework of training evaluation, HRD professionals will prove that the evaluation findings and processes are more 
valid and can be used for important decision-making concerning HRD practices and policies in the organization. 

For the HRD researchers, there must be a challenge to construct and validate the instrument of examining 
the perspectives of stakeholder in the organization on evaluation of training. The proposed framework of training 
evaluation can be used as potential template of the instrument. Furthermore, if the emphasis of evaluation is to meet 
the intents of the stakeholders who will use the information and processes, there must be more needs of 
understanding about match between the intents of the stakeholders and the emphasis of evaluation. In addition, the 
researchers will need to examine the possible conflicting views of stakeholders in the organization because there are 
different outcomes desired by different stakeholders. 
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